Resource-Constrained Non-Operational Testing of Software1

In “classical” testing approaches, “learning” is said to occur if testers dynamically improve the efficiency of their testing as they progress through a testing phase. However, the pressures of modern business and software development practices seem to favor an approach to testing which is very akin to a “sampling without replacement” of a relatively limited number of pre-determined structures and functions conducted under significant schedule and resource constraints. The primary driver is often the desire to “cover” ONLY previously “untested” functions, operations or code constructs, and to meet milestones. We develop and evaluate a model that describes the fault detection and removal process in such an environment. Results indicate that in environments where “coverage”based testing is promoted, but resources and decisions are constrained, very little dynamic “learning” takes place, and that it may be an artifact of program structure or of the test-case sequencing policy.